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ABSTRACT 

Association of Community College Research in 1975, the research 
conducted at community colleges, as well as the issues facing 
researchers, have remained fairly constant. Research has been 
consistently understaffed and the themes of research have tended to 
focus on relating student progress to placement, surveying local 
communities, reviewing programs, and conducting student follow-ups. 
Moreover, the general purposes for conducting institutional research 
have remained constant in that the first purpose is still public 
relations. The need to satisfy external agencies has caused the 
colleges to value anything that makes the school look good and to 
avoid anything that casts a negative light on the institution. 
Institutional researchers must also contend with the need for 
educators to believe what they want to believe. An example of this is 
the frequently reported study showing that problems with students in 
the 1990s have profoundly worsened since the 1940s. Even after the 
study was found to be a hoax, it was continually repeated. Finally, 
in approaching their task, institutional researchers should consider 
the following points: (l) make sure that data is readily 
understandable; (2) understand public relations; (3) employ valid 
methodology; (4) study important questions; (5) perform repeat 
studies; (6) make use of external resources; (7) include only the 
data that is relevant to the research topic; and (8) understand the 
resistance to outcomes data in higher education. (HAA) 



JC 960 530 

Cohen, Arthur M. 

Trends in Institutional Research. 

6 Aug 96 

12p.; Paper presented at the Annual Conference of the 
Southeastern Association for Community College 
Research (25th, Panama City, FL, August 5-7, 

1996). 

Information Analyses (070) — Speeches /Conference 
Papers (150) 

MFOl/PCOl Plus Postage. 

*Communi ty Colleges; *Educat ional Researchers ; 
’'Institutional Research; ’'Politics of Education; 
Program Improvement ; Research Design; ’’'Research 
Problems; Research Reports; Two Year Colleges 

Since the inaugural meeting of the Southeastern 



*****************************yc* ********************************* ******* 

Reproductions supplied by EDRS are the best that can be made 

* from the original document . * 

*********************************************************************** 



o 

ERIC 



Trends in Institutional Research 



<N 

ON 

OO 

ON 



s 



Written By 

Arthur M. Cohen 

ERIC Clearinghouse for Community Colleges 
University of California at Los Angeles 



U.S. DEPARTt^EMT OF EDUCATION 
Office of Educationoi Reaeaich and Impfovamant 

EDUCATIONAL RESOURCES INFORMATION 
r'PMjER 'ERIC! 



This documani has been reproduced as 
>ecaived from the person or organization 
originating it 

Minor changes have bean made to improve 
reproductior^ quality. 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL 

UAS been granted by 

A.M. Cohen 



o Points of view or opinions stated m this docu- 
ment do not necasaorily represent official 
OERl position or policy. 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



Paper presented at the Annual Conference of the Southeastern Association for 
Community College Research (25th, Panama City, FL, August 5-7, 1996). 



O 

UfN 

N© 

tr- 

lERlC 



2 



“Trends in Institutional Research” 

Presentation to Annual Meeting 
of Southeastern Association of Community College Research 
Panama City, Florida 
August 6, 1996 
Arthur M. Cohen 

The Southeastern Association of Community College Researchers is celebrating its 
25 anmversary. Starting in 1981 as a special interest group of the American Educational 
Research Association, by 1975 it had taken on its current identity. I was at its inaugural 
meeting in Boone, North Carolina that year and recall interacting with the founding 
members. In the ensuing years I have had many associations with the group because I am 
the director of the ERIC Clearinghouse for Community Colleges, the national repository 
for institutional research reports. 

ERIC exists to serve you. It provides several supportive activities: it is a source 
for research topics, designs, formats, and questions; it provides a critical review of 
research reports that are sent in for potential entry into the database; it assists researchers 
in writing for publication in any of the journals directed to a community college audience; 
it provides an archive in which researchers may store their reports; and it maintains mailing 
lists of people who are working on various research projects not only in the community 
colleges but also in other institutions and agencies that have some interest in community 
college studies. 

Since the inaugural meeting of SACCR in 1975 numerous research reports have 
come into the ERIC Clearinghouse. Many were written by members of this group and 
others by researchers elsewhere. In reviewing the studies done then and more recently 



some patterns come to light. First, institutional research in the community colleges of the 
nation is as understaffed now as it was 21 years ago. Nationwide the colleges average 
around two-thirds of an FTE in the research office. Second, the number of studies 
emanating from the colleges each year has remained the same. Few new colleges have 
been built and few have merged or closed. Third, the themes of institutional research have 
not changed much. The studies center on relating student progress to placement 
procedures and other variables; surveying the local community together; public 
perceptions of the college; relating student aspiration to curriculum; reviewing programs; 
running comparative studies between the home institution and others, usually in the same 
state; and conducting student follow-up and other types of attainment studies. 

A few things have changed. More college staff members are writing research 
reports. The faculty, the public information director, assistants to the presidents, student 
services staff, program heads, all may be involved in collecting data, running surveys, and 
issuing papers. Frequentiy, the IR director is not consulted regarding study design and 
may not even know of the report until after it has been issued. And second, extramural 
agencies are issuing more demands for data. State agencies have become more prominent 
in requesting information from the colleges. 

In general though, the purposes for institutional research have remained constant. 
The first purpose is still public relations. The research reports form the basis for 
perceptions of the college that are held by members of the public, neighboring institutions, 
and state officials. The need to satisfy external agencies has gained ground as more state 
agencies have mandated that certain data be provided. Far down the list but still worth 
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mentioning as a purpose are the studies that advise program managers on the value of 
their student selection procedures, curriculum, and instructional forms. 

Nonetheless, the IR directors are aware that their emanations are not the only 
influences on college procedures or public perceptions. Public information officials are 
fond of putting out reports about student idiosyncrasies; a student winning an award may 
be worthy of a major release. Administrators will frequently make addresses to the 
college or to the surrounding community based on little or no data or simply on an article 
that they read. The heads of college services such as the bookstore or the cafeteria may 
do their own surveys, catching every Nth person who walks in and asking them how they 
like the service. A program head may survey program graduates asking if they are 
working in the field or whether they have any reactions to the education they received. 

Any of these investigators may use ambiguous questions, poor if any sampling procedures, 
and may draw conclusions that may or may not be based on the data. Even so the reports 
may be treated with as much respect as those prepared by the IR office in which definition 
of the universe, population sampling, question validation, and appropriate statistics are the 
tools of the trade. 

However, this is the milieu in which the IR director functions. The college is 
dependent on the support of its constituency and funding agencies. Therefore, anything 
that makes the college look good is valued whereas anything that casts the institution in a 
negative light is understandably avoided. Sometimes the IR staff gets caught in this need 
to provide a steady stream of good news. As example of the need for a consistent stream 
of positive information is provided in reaction to the Transfer Assembly, a project that the 
Center for the Study of Community Colleges has been operating for seven years. The 
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Center developed a definition of transfer rate that is valid, based on available data, and 
readily understandable; all students entering a community college in the fall of a given 
year, who complete at least twelve credits at the institution, divided into the number of 
that group who matriculate at an in-state public university within four years. Data are 
collected according to that definition from between 15 and 20 states each year. The 
national transfer rate has been holding consistently at around 20% to 22%. California’s 
transfer rate has been hovering between 17% to 19%, consistently lower than the national 
norm. This causes some concern among community college administrators and 
researchers in the state who have acted to shift the focus. First they developed a different 
definition of transfer, computing the transfer rate as the number of students who left the 
institution in good standing in the spring of a given year and did not return in the fall; the 
assumption was that the students might be enrolled elsewhere or at least satisfied with 
their community college experience. A couple of years later they abandoned that and 
defined transfer not in terms of a student’s action but in terms of whether the college had 
made students “ready” for transfer, with transfer readiness defined as the number of 
students who successfully completed the introductory sequence in college level English 
and math, with the assumption that these students were thereby eligible for transfer; 
whether they matriculated at a university or not was irrelevant. All of this distortion just 
as a way of getting out from under the fact that transfer rates in California are slightly 
below the national average. 

The institutional researchers also have to contend with the need for educators to 
believe what they want to believe. A wondrous example of the need to the believe is the 
frequently reported study showing that in the 1940s the major problems with students in 
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the schools were gum-chewing, hair pulling, and running in the corridors whereas in the 
1980s the major problems were drug use, guns on campus, and pregnancy. The New 
York Times Magazine carried a report showing this study to be a hoax, that the 
information about the 1 940s was taken from a survey form that someone was sending to 
school teachers, the information from the 1980s from a focus group of school principals. 
The report even traced the hoax to the person who initiated it. That should have laid the 
matter to rest but because school people want to think that their problems now are 
considerably worse than they were two generations ago, they repeat the story. 

Another example of believing what we want to believe is revealed in the contention 
that bachelor’s degree holders were shut out of California community colleges because of 
a fee increase. Around four years ago the legislature mandated that anyone already in 
possession of bachelor’s degree would have to pay $50 per unit to enroll in a community 
college class whereas people without degrees would pay the standard fee of $13 per unit. 
The college administrators pointed to a sudden drop in enrollments of bachelor’s degree 
holders which contributed to a shortfall of 140,000 students in California the year that the 
extra fee was put in place. However: the students knew that if they checked the 
“bachelor’s or higher” box on the line asking for prior attainment, they would pay $50 a 
unit whereas if they did not check the box their class would cost $13 per unit. Not 
surprisingly the number of bachelor’s degree holders dropped noticeably. But the colleges 
do not verify prior school attainment and the belief that bachelor’s degree holders were 
being shut out, trumpeted repeatedly by the community college spokespersons carried the 
day. The extra fee was remanded three years after it was installed. I think that the 
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proportion of bachelor’s degree holders has since increased significantly among California 
community college students. 

These anecdotes about the need to believe would be only a curiosity except that 
they suggest a way of looking at some of the current beliefs. One is that a tidal wave of 
students will inundate the colleges over the next 5-10 years. This belief is based on data 
showing that the number of 1 8 year olds in the United States will increase notably. And 
certainly the population count is going up such that by 2005 the number of 18 year olds 
will almost be as high as is was in 1979. Therefore, the reasoning goes, the colleges will 
need additional staff and buildings or, if funds are not forthcoming, students will have to 
be diverted to distance learning activities. However, space utilization studies can reveal the 
extent to which this is true. It may just be that there is a considerable amount of slack in 
the system, that scheduling classes more widely throughout the day and the week, 
establishing a year round calendar, and reconfiguring some of the buildings can make 
room for many more students. A study done at Gaston College and reported here 
provides an example. However, the will to believe the colleges must find more resources 
is pervasive. 

Another example of the will to believe is provided in the numerous reports 
showing that community college matriculants are less likely to receive baccalaureate 
degrees than are those students who start as freshmen in a senior institution. Pascarella 
and Terenzini reviewed numerous studies and found most showing that only a small 
percentage of variance in bachelor degree attainment related to the institution where the 
student began. Moreover, for most students, the choice is not between the local 
community college and the freshman class at a university; it is between the community 
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college and no collegiate experience. College leaders should be trumpeting the fact that 
1.25 million students enter the community colleges of the nation each year and that four 
years later 250,000 of them are in baccalaureate-granting institutions; in sum, a quarter 
million potential bachelor’s degree recipients who, absent the community college, might 
not be in the higher education system at all. Instead of that, many higher education 
spokespersons and legislators seize on the lower rate of bachelor’s attainment and draw 
inferences about the poor quality of education that community college students receive. 
They have to believe that their alma mater provided a better experience. 

These issues of the need to believe affect the conduct of institutional research. In 
community surveys for example, questions about whether the public thinks the college is 
doing well and whether they would like to take courses there are less a matter of potential 
student interest than they are public relations. College managers have to believe that the 
public thinks highly of their institution and the IR officer who insists on using validated 
questions, careful population sampling, non-respondent bias check, and similar research 
techniques may be swimming upstream against a strong current. 

The IR directors must also take note that the audience for their reports is not as 
sophisticated in research techniques as they are. Avoid displaying elaborate statistical 
manipulations and second and third-level analysis of marginally relevant data. A petitioner 
deserves straightforward answers to the questions, “How many of your students transfer?” 
or “How many of your students gain employment?” IR must respond with a number 
derived from a valid, readily understandable definition for which data can be obtained. * 
Granted that all sorts of influences not within the college’s ability to control affect the 
numbers and that no one number provides a complete picture. Legislators, newspaper 
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reporters, and few practitioners care to concern themselves with all the reasons or with all 
the ways of partitioning the data. The audience for the college’s releases is not the same 
as a set of research-journal readers. One page is usually enough to report findings. 

Within the broader context of institutional research it is necessary to build a 
constituency. The numerous other college staff members who are conducting studies of 
their own may welcome coordination and advice, provided that they understand it. 
Multivariate analysis is anathema to the history instructors and the psychology instructors 
really don’t need to know that among their students who transferred, the Middle Eastern 
immigrant women aged 45 to 50 with three or more children and a household income of 
$25,000 to $29,000 most appreciated their classes. 

Cooperative ventures are a way of magnifying influence. Nancy Mattice, IR 
director at College of the Canyons (CA) helps the English instructors conduct studies of 
student writing. The faculty receive more reliable data and she gains a receptive audience 
for her reports. Simiiariy the IR directors who conduct studies in concert with their 
counterparts at other colleges, asking the same questions of populations sampled in the 
same way, are able to compare the outcomes of college activities in a way not otherwise 
possible. 
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In summation: 



• Keep it simple. Don’t be surprised that few people are interested in all possible 
permutations and sub-categories of your data. Readily understandable information 
should be the goal. Apply the test, “Would a reasonably literate lay person want to 
know this?” “Would a one-page summary be intelligible?” 

• Understand public relations. The college trustees and administrators have to sustain a 
flow of good news about the college; it’s their job. Hence, studies that yield positive 
findings are valued while anything showing poor results will be submerged unless it 
can be used to Justify additional funding. 

• Do it right. Use best principles of research design; population sampling; non- 
respondent bias check; validated questions. It’s no more difficult and it lends 
credibility to your findings. 

• Study important questions. Comparing the ethnic and gender composition of students 
who take classes at 8:00 am with those who take classes at 9:00 am is Just not very 
useful, whereas analyzing the reasons why some categories of students will not take 
classes at night might inform decisions about campus security. 

• Do repeat studies. Time-series designs that report the same information about the 
college outcomes year after year can reveal the effects of program changes. 
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• Exploit your counterparts. The ERIC files include hundreds of IR studies complete 
with survey forms, and your consortia provide a network for learning which questions 
and methods have proved most efficient in gaining useful information. 

• Leave some data on the table. If you just set out to compare the effects of different 
instructional treatments don’t subdivide the report by student age, gender, ethnicity, 
SES, and aspirations. Yes, those data are readily available. No, they don’t contribute 
to knowledge about instruction; more likely they divert attention from the central 
findings. 

• Know that your audience fears product information. For decades the colleges have 
received their funding on formulas related to student attendance and the staff have 
judged the institution on the basis of its efforts. Higher education has no tradition of 
demonstrating results. Therefore the requests for data on outcomes have been 
shunned, ignored , or responded to with clouds of caveats. Eventually, state-level 
demands such as those recently mandated in Florida and Texas may modify the 
process bias but for now it’s in the air you breath. Be kind to your colleagues. 
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